Skip to content

AsyncAPI 3.x: the fitness, and --problemType ASYNCAPI runs - #1753

Open
LautaroPetaccio wants to merge 12 commits into
masterfrom
feature/asyncapi-fitness
Open

LautaroPetaccio wants to merge 12 commits into
masterfrom
feature/asyncapi-fitness

Conversation

@LautaroPetaccio

@LautaroPetaccio LautaroPetaccio commented Sep 12, 2026 •

Copy link
Copy Markdown
Collaborator

Twelfth in the AsyncAPI stack, on top of #1752. With this one --problemType ASYNCAPI runs a whole search, everything but writing the tests.

What a search over an AsyncAPI service looks for

REST covers (status × endpoint). There are no status codes here, so the targets are:

  • (outcome × operation) — what publishing to an operation was seen to do: PUBLISHED (fire-and-forget), REPLIED, or NO_REPLY.
  • (declared reply × operation) — when a reply came back, which of the messages the contract declares for that reply it was. A reply channel that lists a result and an error gives the search two things to reach, and telling them apart is what makes an asynchronous service observable from outside.
  • (unrecognised reply × operation) — a reply matching none of the declared messages. It is a target of its own, registered independently of the fault below, so the search can still reach that behaviour when the oracles are off.

Two things are faults, and both categories are @Experimental, so they are reported only under --useExperimentalOracles: a promised reply that never arrives, and a reply that matches none of the declared messages. Each is recorded on the action result as well as the fitness, which is where the reports count faults from. A message the driver could not publish is neither — it is a broken setup, no target is registered for it, and the test stops there.

Recognising a reply

AsyncApiReplyClassifier matches the observed payload against each declared payload schema. It is deliberately a classifier, not a validator: it reads the parts of JSON Schema that tell one message from another — type (including type lists), required, const/enum, allOf/anyOf/oneOf, items, $ref to component schemas — and gives the benefit of the doubt on anything it does not understand (formats, bounds, patterns), since a reply it fails to recognise is reported as a fault. When several declared messages fit, the most specific wins: the one pinning down the most top-level fields. No new dependency.

The fitness

AsyncApiFitness resolves everything the driver needs from the document — address, content type, correlation location and pointer, reply address, timeout — so the driver never reads the document. The address comes from the channel's effectiveAddress for the protocol, so a Kafka channel that names no address and carries its topic in bindings.kafka.topic is published to the topic rather than to the channel's name.

The correlation id is runId-counter. runId is a random prefix rather than one drawn from Randomness: a run repeated under the same seed would otherwise reuse the very ids the prefix exists to tell apart, which is the case it is there for. Headers travel as a map by way of the gene's own JSON printing, which is what knows which optional headers are on.

Not supported yet, and said so with a one-time warning: a reply address announced inside the request (reply.address.location) rather than fixed by the contract. Such an operation is published without waiting.

Plumbing

  • RemoteController.executeNewAsyncApiActionAndGetReply, mirroring the RPC call. It has a default that throws, so the four other controllers (three of them test fakes) need not each say they cannot publish.
  • EMConfig.asyncApiReplyTimeoutMs, @Experimental, default 5 s.
  • A constraint: --createTests false is required for AsyncAPI until the test writer exists. Failing at start-up beats searching for an hour and failing at the end. TestCaseWriter is bound to the existing NoTestCaseWriter. Seeding test cases is refused at start-up too, for the same reason. When the problem type is inferred from the driver rather than asked for, test generation is switched off with a warning instead of failing after the SUT has started.
  • AsyncApiModule, one for both modes, binding the driver unconditionally. It does not call super.configure(), like the RPC, GraphQL and Web modules: what EnterpriseModule binds there needs FitnessFunction<RestIndividual>.
  • AsyncApiStructureMutator: one message more or one fewer. Unlike RPC's, it states its two bounds outright — a test keeps at least one message, and can grow to exactly maxTestSize — because mirroring RPC's arithmetic emptied a one-message test when maxTestSize was 2, and never let a test reach the maximum.
  • Main's ASYNCAPI branch binds the module instead of throwing.

Testing

Four suites build the injector Main would, with the driver replaced by a fake, through one shared AsyncApiTestInjector.

  • AsyncApiReplyClassifierTest (20): result vs error behind $refs, undeclared payloads, non-JSON and empty bodies, 3.0 as an integer, 1.0 against 1, most-specific-wins, type lists; and, on a document written for the purpose, const through a chain of references, enum, oneOf/allOf/anyOf, array items, booleans vs their spelling, an unknown type name, and a reference into the middle of a schema being given the benefit of the doubt.
  • AsyncApiFitnessTest (19): every DTO field the driver is told; unique correlation ids across evaluations; result and error as distinct targets; undeclared reply and silence as the two faults, and neither reported without --useExperimentalOracles; an unrecognised reply still covering its own target when they are off; fire-and-forget covered by PUBLISHED; headers as a map without the stamped one; a correlation id located in the payload; a topic from a protocol binding overriding the channel address; the configured timeout; a reply address announced at run time being published without waiting; a driver that could not publish, with and without saying why; a reply without its correlation id recorded rather than judged.
  • AsyncApiStructureMutatorTest (5): the two bounds above, the maximum being reachable, added messages coming from the document, nothing happening when only one message is allowed. The first fails against RPC's arithmetic.
  • AsyncApiModuleTest (2): a whole MIO search, 100 action evaluations, black-box, through the module Main binds, against a stand-in that answers like the NCS service. Asserts that every operation was answered, that some operation reached both its declared replies, that nothing was a fault, and that no correlation id repeated.

All AsyncAPI suites plus the config ones: 205 tests, 0 failures. jacoco over the AsyncAPI package: every new class above 90% line coverage.

@LautaroPetaccio
LautaroPetaccio added this pull request to stack #1712 September 13, 2026 19:50
@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-fitness branch 2 times, most recently from 0cfc38c to a3b119e Compare September 13, 2026 20:52
@arcuri82
arcuri82 force-pushed the feature/asyncapi-fitness branch from a3b119e to 23a9db1 Compare September 14, 2026 10:51
fv.updateTarget(idMapper.handleLocalTarget(fault), 1.0, index)
}

AsyncApiOutcome.PUBLISHED, AsyncApiOutcome.PUBLISH_FAILED -> Unit

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if none of the AsyncApiOutcome matches?

@LautaroPetaccio LautaroPetaccio Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They must all match, as the compiler will not compile otherwise. Did you mean that or were you talking about a reply not matching?

@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-fitness branch 3 times, most recently from 9848d77 to a7856ff Compare September 16, 2026 22:01
@LautaroPetaccio

Copy link
Copy Markdown
Collaborator Author

I forgot to include the replies which were not matched with the schema as possible targets. Now they're included.

@LautaroPetaccio
LautaroPetaccio marked this pull request as ready for review September 18, 2026 21:07
@arcuri82
arcuri82 force-pushed the feature/asyncapi-fitness branch from b961897 to aec5186 Compare September 22, 2026 11:05
* Deliberately not drawn from [randomness]: a run repeated under the same seed would reuse
* the ids it used before, which is the one case this exists to tell apart.
*/
private val runId: String = UUID.randomUUID().toString().take(8)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is problematic since it might lead to different behaviour even when the same -seed is introduced, right?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right! Great catch. I've changed it to use randomness which takes into consideration the seed.

Base automatically changed from feature/asyncapi-sampler to master September 24, 2026 07:27
RemoteController.executeNewAsyncApiActionAndGetReply PUTs the action to the
driver and reads back an AsyncApiReplyDto, as its RPC counterpart does. It
has a default that throws, so the controllers that never publish need not
say so one by one.

EMConfig gains asyncApiReplyTimeoutMs, experimental, and a constraint:
there is no test writer for AsyncAPI yet, so a run must say
--createTests false rather than search for an hour and fail at the end.
The two tests that already parsed --problemType ASYNCAPI say so now.

Two experimental fault categories: a promised reply that never arrives,
and a reply matching none of the messages the contract declares.
AsyncApiBlackBoxFitness publishes each message of a test through the
driver and turns what comes back into targets: for every operation, what
publishing to it was seen to do, and, when a reply came back, which of the
messages the contract declares for the reply it was. A contract listing a
result and an error thus gives the search two things to reach, which is
the AsyncAPI analogue of REST's (status x endpoint).

AsyncApiReplyClassifier is what recognises a reply: a structural match
against each declared payload schema, reading the parts of JSON Schema
that tell one message from another and giving the benefit of the doubt on
the rest. The most specific match wins.

AsyncApiModule binds it all, with the driver bound unconditionally, and
Main uses it in place of the message it showed until now. One test runs a
whole MIO search against a stand-in for the NCS service and sees both
declared replies of an operation covered.
Review: with room for exactly two messages, a one-message test was
mutated by removal, leaving a test that publishes nothing; and a test
could never grow to the maximum the user allowed, only to one less. Both
bounds are now stated as such, and AsyncApiStructureMutatorTest holds
them.

Seeding test cases is refused at start-up, like writing them, instead of
failing inside the sampler. The whole-search suite is named after the
class it exercises, AsyncApiModule, and the three suites that build the
injector share how they do it.
Review: the ':' joining a target id and the '-' inside a correlation id
were spelled out where used; they are constants now, and the two kinds of
target id are built in one place each. The remote controller named its
queryFromDatabase parameter four times over, once in the new call and
three in the ones it mirrors; it is a constant now. The fake driver's one
field comes before its companion.
…aults

Review findings, in order of what they cost a run.

The address a message is published to ignored protocol bindings. A Kafka
channel usually names no address and carries its topic in its binding --
microcks.yaml, already a test resource here, is exactly that shape -- so
messages went to the channel's name and reached nobody. The parser's
effectiveAddress() exists for this and was never called; when a channel
names no address at all, the binding's topic is now taken as the one
destination the document actually gives. An address left holding
{placeholders} is warned about rather than published to silently.

Deciding whether a number is an integer asked Jackson for a BigDecimal,
which throws for a value that overflows a double. Nothing between the
classifier and the search loop catches it, so one odd reply ended the run.

Three ways a valid reply was reported as a fault: a body that is empty or
not JSON, a const or enum written 1.0 against a reply carrying 1, and a
type or combinator this cannot read. A reply with nothing to match against
is now no finding at all, numbers compare by value, and what cannot be read
rejects nothing. A $ref that points into a schema is followed instead of
matching everything, which had quietly disabled the undeclared-reply oracle
for any operation written that way, and specificity counts through the
combinators so the more specific message wins.

Faults ignored isEnabledFaultCategory, so two experimental oracles fired on
a default run and --disabledOracleCodes did nothing. They are recorded on
the action result now, which is where the reports count faults from; before,
a run covered fault targets while every report said zero.

AsyncApiCallResult was the only action result not overriding matchedType.

An inferred problem type no longer dies on the test-generation constraint:
when the driver reports an AsyncAPI service, createTests is turned off with
a warning instead of throwing after the SUT has already been started.

The class is AsyncApiFitness: it is bound as the only fitness and reads
white-box coverage, so BlackBox in its name said something untrue.
The new remote-controller call was a copy of the RPC one; both now share
the body that PUTs an action and reads the reply back.

The structure mutator is a third copy of logic REST and RPC already share,
marked with a TODO the way the GraphQL one marks the same debt.

The reply-timeout description is published verbatim into options.md, where
every neighbour is one clause; the reasoning behind the option belongs in
the pull request, not in the user's terminal.
The guards that keep a valid reply from being reported as a fault had no
tests, which is why they could be wrong. Now covered: a reply with no body,
an operation whose reply declares no message, a number too large to be a
decimal, a const compared by value, a reference into a schema, and the most
specific match found through a combinator.

Also covered: the topic a protocol binding names, the configured reply
timeout, and that faults stay quiet until experimental oracles are asked
for -- each of which would otherwise pass with the code deleted.

The structure mutator's tracking is exercised with a real specification
rather than null, so the add/remove bookkeeping the archive reads is
checked. The module test asserts the bindings Main resolves, since this
module deliberately inherits none. The sampler suite resets the static gene
cache its siblings all reset.
The reply DTO no longer uses primitives, so a field the driver never set
arrives as null rather than as a default. Kotlin maps those to platform
types, so reading them as before would still compile and throw at run time.

Each is now read deliberately. Whether the message was published has no
safe default, so an absent answer is treated as a failure and said so in
the error message, apart from a driver that answered false. Whether a reply
was expected or arrived is read as not having happened when unset, which is
what a fire-and-forget driver leaves blank.

Correlation is the one that changes behaviour: a driver that does not track
it at all reported false, which reads as the service having failed to echo
the id back. That is a claim about the SUT the driver never made, so it is
now recorded only when the driver actually looked.
Turning the experimental oracles off used to cost coverage rather than only
reporting. A reply matching none of the declared messages registered nothing
beyond the gated fault call, so it was indistinguishable from a recognised one
and could be dropped when the solution was minimised. It now covers a target of
its own, registered before the fault and independent of whether it is enabled,
which is how a 500 is already handled for REST.

Also in this round:

- the correlation id prefix no longer comes from the seeded generator, which
  made a run repeated under the same seed reuse the very ids it exists to tell
  apart
- an outcome that cannot reach target handling now says so by throwing rather
  than by a comment, and the branch stays exhaustive so that a newly added
  outcome fails to compile instead of falling through
- why there is no separate black-box fitness, and why the reply classifier is
  written by hand rather than delegated to the validator already on the
  classpath, which reads draft-04 and so does not know const
The driver contract now names the two places a correlation id can travel
with an enum rather than a pair of string constants, so the fitness sets
the field with the enum and the tests compare against it.
@arcuri82
arcuri82 force-pushed the feature/asyncapi-fitness branch from 315c227 to 5911a41 Compare September 24, 2026 07:27
The run prefix was drawn outside the seeded generator, so two runs under
the same seed published different correlation ids. Nothing in the search
reads that value today, but the guarantee that a seed reproduces a run
should not rest on nobody reading it, and where the contract puts the id
in the payload the published bytes already differ.

It is back on the generator, and the reason it had left is written down
where it can be enforced: a reply published before the action was is not
an answer to it, and only the driver can ensure that, by seeking to the
end of the reply destination or by a fresh subscription. The reference
Kafka driver already does, which is why the prefix never needed to.

A test publishes the same two messages twice under one seed and compares
the ids; it fails against the previous value.
@jgaleotti
jgaleotti requested a review from arcuri82 September 24, 2026 18:59
* before the gene builder sees it. A validator that reads a modern draft would replace most of
* this, at the cost of a new dependency.
*/
object AsyncApiReplyClassifier {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a utility function should not be under a service package (which is for module declarations and singletons)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved it to org.evomaster.core.problem.asyncapi.classifier along with its tests.

} else if (info.asyncApiProblem != null) {
config.problemType = EMConfig.ProblemType.ASYNCAPI
if (config.createTests) {
/*

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leave a TODO message stating that till will need to be removed in the future

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

It is a stateless object nothing injects, sitting beside the sampler,
fitness, module and mutator, which are all Guice-managed. It moves to a
package of its own, with its test.

Main also says now when the block that switches test generation off has
to go, rather than only why it is there.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants